Lecture 18

None of these models fits the data well.

\[ \begin{aligned} \mathcal{M}_1 &= \qty{ m(w,x) = a + bx \ : \ \text{ any numbers }\ a,b } && \text{ one line } \\ \mathcal{M}_2 &= \qty{ m(w,x) = a(w) : \ \text{ any function }\ a(w) } && \text{ horizontal lines } \\ \mathcal{M}_3 &= \qty{ m(w,x) = a(w) + bx \ : \ \text{ any function }\ a(w) \ \text{ and number }\ b } && \text{ parallel lines } \\ \mathcal{M}_4 &= \qty{ m(w,x) = a(w) + b(w)x \ : \ \text{ any functions }\ a(w),\ b(w) } && \text{ lines } \\ \mathcal{M}_5 &= \qty{ m(w,x) = a(w) + b \operatorname*{1_{\le 65}}(x) \ : \ \text{ any function } a(w) \ \text{ and number }\ b } && \text{ additive piecewise-constant } \\ \mathcal{M}_6 &= \qty{ m(w,x) = a(w) + b(w) \operatorname*{1_{\le 65}}(x) \ : \ \text{ any functions } a(w),\ b(w) } && \text{ piecewise-constant } \end{aligned} \]

But one does give us an unbiased estimate of the ATT.

Why?

• Let’s think about what we’re estimating: the ATT.
• This is a weighted average of column means.

\[ \begin{aligned} \text{ATT} &= \frac{1}{m_1}\sum_{j:w_j=1} \qty{ \mu(1,x_j) - \mu(0,x_j)} \qfor \mu(w,x) = \frac{\sum_{j: w_j,x_j=w,x } y_j}{\sum_{j: w_j,x_j=w} 1} \qand m_w = \sum_{j: w_j=w} 1 \\ &= \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x)} \qty{ \textcolor[RGB]{0,191,196}{\mu(1,x)} - \textcolor[RGB]{248,118,109}{\mu(0,x)}} \qfor p(w,x) = \frac{\sum_{j: w_j=w, x_j=x} 1}{\sum_{j: w_j=w} 1 } \\ &= \sum_{w,x} \alpha(w,x) \mu(w,x) \qfor \alpha(w,x) = \begin{cases}+p(1,x) & \text{ if } w=1 \\ -p(1,x) & \text{ if } w=0 \end{cases} \end{aligned} \]

• What you may notice is that, because these proportions \(p(w,x)\) are piecewise-constant, they’re in the model.

A Claim

• You’ll get an unbiased estimate of the summary \(\sum_{w,x}\alpha(w,x) \mu(w,x)\)
• if the regression model \(\mathcal{M}\) contains at least one of …
  • the subpopulation mean function \(\mu(w,x)\)
  • the weights \(\alpha(w,x)\)

Proof Ideas

1. Remember that our sampling distribution’s mean is our summary with the population least squares fit \(\tilde\mu\) plugged in for \(\mu\). \[ \mathop{\mathrm{E}}\qty[\widehat{\text{ATT}}] = \widetilde{\text{ATT}} := \textcolor[RGB]{0,191,196}{p(1,x)} \qty{ \textcolor[RGB]{0,191,196}{\tilde \mu(1,x)} - \textcolor[RGB]{248,118,109}{\tilde \mu(0,x)}} \]

2. Think about what we know about population least squares residuals.

\[ \begin{aligned} \frac{1}{n}\sum_{i=1}^n \qty{Y_i - \hat\mu(W_i,X_i)} m(W_i,X_i) &= 0 \qqtext{ for every } m \in \mathcal{M}&& \text{\textcolor{gray}{(sample least squares residuals)}} \\ \mathop{\mathrm{E}}\qty[ \qty{Y_i - \tilde\mu(W_i,X_i)} m(W_i,X_i)] &= 0 \qqtext{ for every } m \in \mathcal{M}&& \text{\textcolor{gray}{(population least squares residuals)}} \\ \end{aligned} \]

Weighted Least Squares

• This sort of feels like an ‘a stopped clock is right twice a day’ thing.
• We chose a ‘bad model’ for \(\mu\) but it worked because it was a good model for \(\alpha\).
• But we can make this more systematic by using weighted least squares.

\[ \begin{aligned} \hat \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \frac{1}{n}\sum_{i=1}^n \textcolor[RGB]{0,0,255}{\gamma(W_i,X_i)} \qty{Y_i - m(W_i,X_i)}^2 \quad \text{ satisfies } \quad &\frac{1}{n}\sum_{i=1}^n \textcolor[RGB]{0,0,255}{\gamma(W_i,X_i)} \qty{Y_i - \hat\mu(W_i,X_i)} m(W_i,X_i) = 0 && \qqtext{ for every } m \in \mathcal{M}\\ \tilde\mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \mathop{\mathrm{E}}\qty[ \textcolor[RGB]{0,0,255}{\gamma(W_i,X_i)} \qty{Y_i - m(W_i,X_i)}^2 ] \quad \text{ satisfies } \quad &\mathop{\mathrm{E}}\qty[ \textcolor[RGB]{0,0,255}{\gamma(W_i,X_i)} \qty{Y_i - \tilde\mu(W_i,X_i)} m(W_i,X_i)] = 0 && \qqtext{ for every } m \in \mathcal{M} \end{aligned} \]

• Our weights should reflect our summary.
• e.g. if we take \(\gamma(w,x) = \abs{\alpha(w,x)}\), we’re good if our model contains \(m(w,x) = \text{sign}(\alpha(w,x))\).
• For treatment effects, that means — if we choose the right weights — we just need to have indicators for treatment.

Inverse Probability Weighting

• This just sort of worked out for us ‘by chance’ with our block-randomized experiment.
• But let’s think about how to get the right weights in general. What are we doing?
• We’re averaging the the red and green curves over the distribution of the green points.

\[ \]

Does the Matched Term Cause Any Trouble?

\[ \text{bias} = \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x) \qty{\tilde\mu(1,x) - \mu(1,x)}} \textcolor{lightgray}{ - \sum_{x} p(1,x) \qty{\tilde\mu(0,x)-\mu(0,x)}} \]

• If our model has treatment-group indicators, no.

\[ \begin{aligned} 0 &= \mathop{\mathrm{E}}\qty[ \qty{Y_i - \tilde\mu(W_i,X_i)} 1(W_i=1) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(W_i,X_i) - \tilde\mu(W_i,X_i)} 1(W_i=1) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(1,X_i) - \tilde\mu(1,X_i)} 1(W_i=1) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(1,X_i) - \tilde\mu(1,X_i)} \mid W_i=1 ] P(W_i=1) \\ &=\sum_{x} \textcolor[RGB]{0,191,196}{p(1,x) \qty{\mu(1,x) - \hat\mu(1,x)}} \times P(W_i=1) \end{aligned} \]

• Intuitively, this is because we’re ‘fitting the data in the right place’
• We’re averaging over the same distribution of points we’ve fit our curve to.
• So, even if our model is ‘bad’ most places, its average is still right.

What About the Mismatched Term?

\[ \text{bias} = \textcolor{lightgray}{\sum_{x} \tilde p(1,x) \qty{ \tilde\mu(1,x) - \mu(1,x)} -} \sum_{x} \textcolor[RGB]{0,191,196}{\tilde p(1,x)} \textcolor[RGB]{248,118,109}{\qty{\tilde\mu(0,x) - \mu(0,x)}} \]

• Even when our model has treatment-group indicators, it’s still trouble. Let’s try what we just did.

\[ \begin{aligned} 0 &= \mathop{\mathrm{E}}\qty[ \qty{Y_i - \tilde\mu(W_i,X_i)} 1(W_i=0) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(W_i,X_i) - \tilde\mu(W_i,X_i)} 1(W_i=0) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(0,X_i) - \tilde\mu(0,X_i)} 1(W_i=1) ] \\ &=\mathop{\mathrm{E}}\qty[ \qty{\mu(0,X_i) - \tilde\mu(0,X_i)} \mid W_i=0 ] P(W_i=0) \\ &= \sum_{x} \textcolor[RGB]{248,118,109}{p(0,x) \qty{\mu(0,x) - \tilde\mu(0,x)}} \times P(W_i=0) \end{aligned} \]

• This is not the right average. It’s the one that occurs in the ‘unadjusted gap’, not the ATT.
• We need to adjust for the fact that we’re averaging over a different distribution of points.

The Second: A Weighted Version

\[ \text{bias} = \textcolor{lightgray}{\sum_{x} \tilde p(1,x) \qty{ \tilde\mu(1,x) - \mu(1,x)} -} \sum_{x} \textcolor[RGB]{0,191,196}{\tilde p(1,x)} \textcolor[RGB]{248,118,109}{\qty{\tilde\mu(0,x) - \mu(0,x)}} \]

• Recall what we know about weighted least squares and its residuals. \[ \tilde \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \mathop{\mathrm{E}}\qty[ \gamma(W_i,X_i) \qty{Y_i - m(W_i,X_i)}^2 ] \quad \text{ satisfies } \quad \mathop{\mathrm{E}}\qty[ \gamma(W_i,X_i) \qty{Y_i - \tilde\mu(W_i,X_i)} m(W_i,X_i)] = 0 \qqtext{ for every } m \in \mathcal{M} \]

• Let’s try ‘upweighting’ the red points on the right side by the ratio of the number of green points to red ones.

\[ \begin{aligned} 0 &= \mathop{\mathrm{E}}\qty[ \gamma(X_i) \qty{Y_i - \tilde\mu(W_i,X_i)} 1(W_i=0) ] \qfor \gamma(x)= \frac{\textcolor[RGB]{0,191,196}{p(1,x)}}{\textcolor[RGB]{248,118,109}{p(0,x)}} \\ &=\mathop{\mathrm{E}}\qty[ \gamma(X_i)\qty{\mu(W_i,X_i) - \tilde\mu(W_i,X_i)} 1(W_i=0) ] \\ &=\mathop{\mathrm{E}}\qty[ \gamma(X_i)\qty{\mu(0,X_i) - \tilde\mu(0,X_i)} 1(W_i=1) ] \\ &=\mathop{\mathrm{E}}\qty[ \gamma(X_i)\qty{\mu(,X_i) - \tilde\mu(0,X_i)} \mid W_i=0 ] \textcolor{lightgray}{P(W_i=0)} \\ &= \sum_{x} \textcolor[RGB]{248,118,109}{p(0,x)} \gamma(0,x) \textcolor[RGB]{248,118,109}{\qty{\mu(0,x) - \tilde\mu(0,x)}} \textcolor{lightgray}{\times P(W_i=0)} \\ &= \sum_{x} \textcolor[RGB]{248,118,109}{p(0,x)} \frac{\textcolor[RGB]{0,191,196}{p(1,x)}}{\textcolor[RGB]{248,118,109}{p(0,x)}} \textcolor[RGB]{248,118,109}{\qty{\mu(0,x) - \tilde\mu(0,x)}} \textcolor{lightgray}{\times P(W_i=0)}\\ &= \sum_{x} \textcolor[RGB]{0,191,196}{p(1,x)} \textcolor[RGB]{248,118,109}{\qty{\mu(0,x) - \tilde\mu(0,x)}} \textcolor{lightgray}{\times P(W_i=0)} \end{aligned} \]

• This works. Intuitively, using these weights means that we’re ‘fitting the data in the right place’ again.

Summary

\[ \begin{aligned} \widehat{\text{ATT}} = \sum_{x} \textcolor[RGB]{0,191,196}{\hat p(1,x)} \qty{ \textcolor[RGB]{0,191,196}{\hat\mu(1,x)} - \textcolor[RGB]{248,118,109}{\hat\mu(0,x)}} \qfor \hat \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \sum_{i=1}^n \gamma(W_i,X_i) \qty{Y_i - m(W_i,X_i)}^2 \qfor \frac{\textcolor[RGB]{0,191,196}{p(1,x)}}{p(w,x)} \end{aligned} \]

• This is the ‘inverse probability weighted least squares’ estimate of the ATT.
• We give equal weight to the green points because that’s the distribution we’re averaging over.
• But we weight the red points by the ratio of the number of green points to red ones so that …
  • if we ‘count them by weight’ …
  • then they look like they have that distribution too.

Summary

\[ \begin{aligned} \widehat{\text{ATT}} = \sum_{x} \textcolor[RGB]{0,191,196}{\hat p(1,x)} \qty{ \textcolor[RGB]{0,191,196}{\hat\mu(1,x)} - \textcolor[RGB]{248,118,109}{\hat\mu(0,x)}} \qfor \hat \mu = \mathop{\mathrm{argmin}}_{m \in \mathcal{M}} \sum_{i=1}^n \gamma(W_i,X_i) \qty{Y_i - m(W_i,X_i)}^2 \qfor \frac{\textcolor[RGB]{0,191,196}{p(1,x)}}{p(w,x)} \end{aligned} \]

This Works For Any Distribution of Assignments